Subject: Revised Flow Control Proposal

From: [email-masked]
Date: Tue, 22 Feb 94 15:07:01 EST

I revised my 1/27 proposal in accordance with what I understand to be
the consensus positions reached in last week's RIG meeting.  Here's
the new version.  We can resolve any discrepancies with the minutes when
they come out, but I think the details are fair game for discussion
in any case.  I will send a later note with example flows (as was
requested in the meeting) and a few points for further discussion.
Peter

=============================================
DLSw FLOW CONTROL PROPOSAL (2/22/94 Version):
=============================================
(Numbers in parentheses refer to the NOTES section which follows)

PRE-STANDARD FLOW CONTROL - (1)
===============================
DLSw nodes which do not support capabilities exchange may be assumed
to have the following flow control support:
  The node can receive ENTER_BUSY and EXIT_BUSY and will stop and start
  (respectively) sending I-frame traffic to the x_BUSY originator on the
  specified circuit.  The node may also send ENTER_BUSY and EXIT_BUSY.
  The node supports x_BUSY messages at least in the circuit state
  CONNECTED (1a), and does not run pacing on circuits using x_BUSY.
The trigger conditions for sending x_BUSY and exactly how fast a node reacts
to x_BUSY are implementation-defined.  Some nodes may buffer frames,
while others may simply use these signals to directly reflect data link
RR/RNR conditions.

Standard DLSw nodes which wish to interoperate with pre-standard nodes
must be able to:
  - Discern a partner node's pre-standard status.
  - Assume the function of a pre-standard node for circuits with that
    partner, as defined above.  Having the capability to send x_BUSY
    is not required.
When using x_BUSY on connections with pre-standard nodes, a standard
DLSw node will simultaneously use pacing on connections with other
standard nodes.

CAPABILITY EXCHANGE -
=====================
DLSw nodes which perform capabilities exchange must implement the
send and receive sides of the adaptive pacing logic described below,
on a circuit basis.  Pacing receivers are not obligated to ever vary
the pacing window size, but the sender must be able to respond correctly
to window size changes (both solicited and unsolicited). (2)

As part of capabilities exchange, each node reports to the other the
initial pacing window size it is expecting in its role as a traffic
receiver.  This control vector is required;  it must be both supplied
and used.  Failure to include it is a capabilities exchange protocol
error and results in transport connection termination.(3)  The two
window sizes are independent;  each node is obligated to begin as a
sender with the initial window size passed by its partner node.

ADAPTIVE PACING -
=================
Flags/fields in the short ("information header"), 2nd byte:
- Pacing Request            (PREQ) - 1 bit
- Pacing Response           (PRSP) - 1 bit
- Increment Window Indicator (IWI) - 1 bit
- Halve Window Indicator     (HWI) - 1 bit
- Reset Window Indicator     (RWI) - 1 bit

New SSP message types: (4)
IPM - simply the message type in the 16-byte information message header,
      with no fields following the header

Function:
Pacing is done on a circuit basis only, in both directions independently.
The pacing receiver controls via a sliding cumulative window how many
frames a sender may put on the transport connection, for a given circuit.
Pacing is applied only to SSP message types INFOFRAME and DGRMFRAME.  The
point at which a DLSw node should RNR a data link or begin to discard
UI-frames is an implementation issue for the router and is not specified
by this pacing architecture.

The sender sets PREQ in the first frame of every window.(5)  It may continue
to send the full window's worth of frames before it must stop and await a
pacing response.  When a pacing response is received, the sender adds the
current next window size (as adjusted by the just-received IWI or HWI) to
the current remaining window size to reach a new current window size.
The initial next window size is the same as the initial current window
size, determined during capability exchange.

In normal operation, the receiver receives I/UI-frames and looks for
a frame with PREQ set.  When it sees one, it determines based on
implementation-dependent conditions how to adjust the next window size and
when to send the pacing response (it could be withheld).(6)  The receiver
controls the window sizes it grants, and is not required to pre-allocate
buffer space for all the outstanding windows.

Solicited pacing responses (those sent in response to PREQ) may be sent
piggy-backed or as IPMs.  In either case, the pacing receiver sets PRSP
and one or neither of IWI and HWI.  Setting neither leaves the next window
size at its current value.  Setting IWI increments it by one, and setting
HWI sets it to MAX(FLOOR(current NWS / 2),1).

To respond quickly to congestion, the pacing receiver may send an unsoli-
cited "reset window" indication to the pacing sender.  This indication may
be sent only as an IPM (not piggy-backed), with PRSP=0 and RWI=1.  Upon
receipt of this message, the sender immediately ends the current window and
sets the next window size to 0.  The sender must immediately reflect the
RWI bit along with PREQ in an IPM it sends to the receiver, both to
solicit a new window and to allow the receiver to determine the end of
the window outstanding at the time it sent the RWI.  The initial IPM with
RWI resets a "PRSP owed" condition at both the pacing sender and receiver,
but the receiver must still treat the entire currently granted window as
outstanding until it receives PREQ+RWI back.

The pacing receiver may in fact send an IPM with RWI in place of a regular
solicited pacing response, since it resets a "PRSP owed" condition.

MANAGEMENT CONSIDERATIONS -
===========================
The following are suggested fields for the "standard" DLSw MIB:
- whether this node supports connections with pre-standard nodes (get)
- as part of the partner list, the partner node DLSw level (standard
  or pre-standard implies FC type) (get)
- as part of the circuit list, the following values being maintained
  as a pacing sender:  current pacing window, current next window size (get)
- administrative state to cause circuit reset (as the pacing receiver) (set)

NOTES:
======
1) Since we decided to put pre-standard considerations in their own
   section in the new RFC, here's the flow control piece of that.  In
   choosing to refer to I-frames only and not require immediate RR/RNR,
   this section reflects discussions I had last week with Ed Stern of
   Proteon and Scott Barvick of Wellfleet.  If any other pre-standard
   vendors feel ill-served by these choices, please speak up now!
1a) I'm still checking on the question Paul Brittain raised, about the
   other states in which the 6611 supports x_BUSY.
2) A receiver may drive a circuit in a "fixed pacing" mode by:
   - never setting IWI or HWI in solicited pacing responses
   - never sending unsolicited indications with RWI=1.
   In this mode, the initial pacing window size and next window size at
   the sender are always the same constant value set by the receiver
   during capabilities exchange.  The only circuit-level mechanism a
   pacing receiver has to slow down a sender is to withhold pacing
   responses.
3) I assume the capex RIGlet will define a reject code for "pacing CV
   not included", or maybe just "required CV not included".
4) Alan Bartky will need to assign a new SSP code point for IPM.  IBM
   was originally using x'7C' for IPM in the previous incarnation of
   this pacing scheme, if that's still available.
5) APPN does have this rule for ISR pacing.  Old SNA allowed it later in
   the window, as many of you informed me, but products ditched this idea
   long ago (the "Jurassic" period, I was told!) when it was found to
   have no value.  Of course we could change this if we need the window
   position to carry some RLWI information.
6) The receiver makes this determination with no knowledge of how well
   the sender is using the current window (there is no "request larger
   window" indicator).  This is a weakness of this scheme, and several
   people are working on how to get around it.  Paul Wood should be
   making a suggestion in a few weeks.

---

From: Bernard Brady <[email-masked]>
Date: Tue, 8 Mar 94 12:22:45 EDT

Peter,

Overall the proposal looks great, but I have a question on the following
statement.

>by this pacing architecture.

I think that this section should be re-worded as follows:

	Pacing is done on a circuit basis only, in both directions
	independently. The pacing receiver controls via a sliding
	cumulative window how many frames a sender may put on the
	transport connection, for a given circuit. Pacing is applied
	only to SSP message type INFOFRAME.  The point at which a DLSw
	node should RNR a data link or begin to discard UI-frames is an
	implementation issue for the router and is not specified by this
	pacing architecture.

I remove the DGRMFRAME, if not then  the last sentence conflicts.

Bernie

---

From: Bernard Brady <[email-masked]>
Date: Thu, 10 Mar 94 13:03:33 EDT

Peter,

>>ADAPTIVE PACING -

Since this will also be set in  DGRM frames this byte will also be
required in the long header also. Since this is also the case for IPM's
it make since to use the same byte and definitions in all cases.  By
what is specified the ADAPTIVE PACING byte is only specified in the
short header.  This should be doumented for both types.

I would also clarify that Pacing is done only on circuits in the
Connected state,  I take it that  DGRMFRAME was used to imply this.

The only problem that I see is that while in the connected state the FSM
specifies that a DLSw can receive either a DGRMFRAME or a DATAFRAME. 
Since they are both the same thing one will be paced and the other will
not.  If the pacing window was closed and a DATAFRAME is received then
it does not violate the PACING protocol. I don't know if this creates an
issue or an implementation benefit.  

Error Cases:

One other thing that is missing from the current write up is error
recovery.  If a pacing widow is exceeded then what is the action to be
taken.  My take is to send a HALT_DL_NOACK.  I have included a flow 
which is using a  window size of 7.  From the network DLSw receives
INFOFRAMES.  The IPM is held up by DLSw because  DLC has reported  a
busy condition.   On receiving the 8th frame DLSw considers the pacing
widow to be violate and issues a HALT_DL_NOACK to the DLSw peer.

Flow Chart:
 
	DLC			   	DLSw			NETWORK
			  |							|
			  |	[CONNECTED]			|
			  |							|
			  |			      <-:-----INFOFRAME(1)
			<-:------INFO					|
			  |			      <-:-----INFOFRAME(2)
			<-:------INFO					|
			  |			     <-:-----INFOFRAME(3)
			<-:------INFO					|
			  |			      <-:-----INFOFRAME(4)
			<-:------INFO					|
			  |			      <-:-----INFOFRAME(5)
			<-:------INFO					|
	DLC_DL_BUSY ------:->				|
			  |			      <-:-----INFOFRAME(6)
			<-:------INFO					|
			  |			      <-:-----INFOFRAME(7)
			<-:------INFO					|
			  |			      <-:-----INFOFRAME(8)
			<-:------HALT_DL				|
			  |							|
			  |		HALT_DL_NOACK-------:->
			  |							|
			  | [HALT_PENDING_NOACK]	|
			  |				       			|
			  |			      <-:-----INFOFRAME
	DLC_DL_HALTED ----:->				|
			  |				        		|
			  |	[DISCONNECTED]		|
			  |